BRANCHING COEFFICIENTS OF HOLOMORPHIC REPRESENTATIONS 
AND SEGAL-BARGMANN TRANSFORM 



GENKAI ZHANG 

Abstract. Let ED = G/K be a complex bounded symmetric domain of tube type in a 
Jordan algebra Vfc, and let D = H/ L = ED n V be its real form in a Jordan algebra V C Vc- 
The analytic continuation of the holomorphic discrete series on ED forms a family of inter- 
esting representations of G. We consider the restriction on D of the scalar holomorphic 
representations of G, as a representation of H. The unitary part of the restriction map 
gives then a generalization of the Segal-Bargmann transform. The group L is a spherical 
subgroup of K and we find a canonical basis of L-invariant polynomials in components of 
the Schmid decomposition and we express them in terms of the Jack symmetric polyno- 
mials. We prove that the Segal-Bargmann transform of those L-invariant polynomials are, 
under the spherical transform on D, multi-variable Wilson type polynomials and we give a 
simple alternative proof of their orthogonality relation. We find the expansion of the spher- 
ical functions on D, when extended to a neighborhood in D, in terms of the L-spherical 
holomorphic polynomials on ED, the coefficients being the Wilson polynomials. 



1. Introduction 

Let D = G/K be a complex bounded symmetric domain of tube type. The weighted 
Bergman spaces Ti. u on D form unitary representations of G and are also called the scalar 
holomorphic discrete series. They have analytic continuation in terms of the weight v and 
constitute some interesting and important family of unitary representations of G. It has 
turned out that it is very fruitful to study the restriction of the holomorphic representation 
to certain subgroups, both from the point of representation theory and harmonic analysis. 
In this paper we will pursue this by studying the restriction on some real forms of D. 

The domain D can be realized as a unit ball in a Jordan triple Vc- Let V be a real form of 
Vc, Vc = V + iV. The real form D = V H © is called a real bounded symmetric domain 
if the complex conjugation r with respect to V keeps D invariant. In this case D = H/L 
is also a Riemannian symmetric space where H is a symmetric subgroup of G and L is a 
symmetric subgroup of K. Thus we have the following commutative diagram of subgroup 
inclusions 

G < K 



H 



Key words and phrases. Holomorphic discrete series, highest weight representations, branching rule, 
bounded symmetric domains, real bounded symmetric domains, Jordan pairs, Jack symmetric polynomials, 
orthogonal polynomials, Cayley identity. 
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We consider the branching law of the holomorphic representation 7i v on D along the 
diagram. The branching of Ji v under K is given by the Schmid decomposition, whereas its 
restriction to H (the left vertical line) is given by the generalized Segal-Bargmann transform 
(see [ j2~6| ] and [^9|]), which gives the unitary equivalence between H u as the L 2 -space on the 
D. To continue this study, we consider the L-invariant elements in Ji v . The branching 
along the lower horizontal line is then given by the Helgason spherical transform; so to get 
diagram around we need to find all the L-invariant elements in the Schmid decomposition 
and calculate their Segal-Bargmann and spherical transform, this will be the main task of 
the present paper. 

Let r be the rank of D, so that the rank of D is also r if the root system of H/L is 
of type A r -i or D r , it is 2r other wise; see Appendix 2. Let W be the Weyl group of 
the root system of the symmetric space H/L. The space V = V(Vc) of holomorphic 
polynomials on Vc under the natural action of the compact group K is decomposed into 
irreducible subspaces Vn with signature n = n 1 7 1 + • • • + n r ^ r for types A, D or B, 
and n = n^x + n'fli + • • • + n r ^ r + n' r ^' r for other types, with multiplicity one. This 
decomposition can also be viewed as the diagonalization of a system of Cayley-Capelli 
type operators, namely A(z)~ a A(d) A(z) a+1 , where A is the determinant polynomial of 
the Jordan triple Vc and a are nonnegative integers; moreover the eigenvalues of those 
operators separate the spaces Vn- We consider its subspace V^ of L-invariant elements. 
By using the Cartan-Helgason theorem we find those signatures n, to be called spherical 
signature, for which V^ ^ 0. Our first goal will be to find those polynomials. 

For type A those polynomials are the well-known Jack symmetric polynomials and their 
norm has been calculated by Faraut-Koranyi [|7|]; there are also studied intensively by com- 
binatorial method p3|]. We shall study in |t3]] their Segal-Bargmann transform both in 
bounded and unbounded realization in relation to the Laplace-transform. However for other 
types those polynomials have not been determined in representation theory. We will use 
the Chevalley restriction theorem and the Dunkl-Cherednik operators to find them. 

Let f) = q + [ be the Cartan decomposition of the Lie algebra h of if and let a be a 
maximal abelian subspace of q. Denote £ the root system of (h, o) and W the correspond- 
ing Weyl group. The space q can be identified with the Jordan triple V. By a well-known 
theorem of Chevalley the restriction map from V L of L-invariant polynomials on Vc to a is 
an isomorphism onto the subspace V(a) w of VF-invariants polynomials in V(a). Thus for 
each L-spherical signature n there exists up to constants a unique polynomials in Vn- Let 
L>£, £ G o, be the Dunkl operators. We can construct a family Ui of commuting operators 
acting on polynomials on a. For root system of type A, C those operators have been previ- 
ously constructed by Dunkl [Kp, We prove that when acting on L-invariant polynomials p 
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and restricted to o, 

r 

A(z)- a A(d)A(zy +1 p(z) = Y[{Ui + a)p(z). 

i=i 

Thus the problem of diagonalizing the Cayley-Capelli operator is reduced to that for the 
operators C/j. We find those polynomials p R E in terms of the Jack polynomials and we 
calculated the Fock space norm of those polynomials; for type C those are done in by 
some different method. We can then find the norm of those polynomials in the Bergman 
space 7i p , by using the result of Faraut-Koranyi. 

We calculate then the Fourier and spherical transforms of the Segal-Bargmann trans- 
forms of those polynomials in the setting of Fock spaces and respectively Bergman spaces. 
They are, up to a factor of the square root of the symbol of the Berezin transform, Weyl 
group invariant orthogonal polynomials and will be called the branching or coupling co- 
efficients as appeared in the title. In the former case we prove that they are of the form 
e~4?ll-H 2 Cn,i/(A), where Cn,i>(A) are Hermite type polynomials. Let J v (x, A) be the Bessel 
function associated with the action of L on V, we prove that in the expansion 
in terms of the coefficients are exactly Cn,i/(A). In the later case (curved case) the 

spherical transform of the Segal-Bargmann transform of p„ is of the form &j,(A)5£n(A), 
where fe„(A) is the symbol of the Berezin transform; the symbol has been found indepen- 
dently in [|34|], [[Z4p and J39|]. (See also [ f32| ] for the case of complex bounded symmetric 
domains.) Thus the polynomials £m(A) are W -invariant orthogonal polynomial with re- 
spect to the measure b u (\)\c(\)\~ 2 where c(A) is the Harish-Chandra c-function. They are 
some limiting cases of the multi- variable Wilson polynomials [p3[], so that the measure 
in the orthogonality relation has now an analytic significance. We find also an expan- 
sion of the product of the spherical function on D with the reproducing kernel in terms of 
the L-invariant polynomials, the coefficient being the spherical transform of their Segal- 
Bargmann transform (see Theorem 9.1). We give thus a unification of the two types of 
orthogonal polynomials associated to a root system, homogeneous symmetric polynomials 
of Jack type on one hand and the non-homogeneous Wilson polynomials on the other. For 
a general root system the Wilson polynomials have been studied by van-Diejen p3[ ] under 
certain self-dual condition. 

Along our way of the study we find an isometric version of the Chevalley restriction 
theorem (see Proposition 5.2) and an analogy of the Capelli identity expressing the product 
Y\ Xj and Y[ Dj in terms of the Cherednik operators Y[ Uj, which we believe are also of 
independent interest; see [ |31| ] and [[15]] for the related study. 



We remark that Theorem can be deduced from [Q], provided that one proves that 
they are eigenpolynomials of the (product of) the Cherednik operators and identify ours 
with that of Dunkl; in that paper Dunkl studied the a more general class of polynomials 
invariant under certain subgroups of the Weyl group and found the norm. 
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We refer also the reader to [[Bp for some algebraic consideration about finding polyno- 
mial invariants of linear groups and [[T7[], [ |T%[ ] for some general results concerning the 
branching of unitary highest weight representations. We mention also that our results 
can also be interpreted as finding eigenfunctions of the Hamiltonians for the Calogero- 
Sutherland model for many body system, both with rational (flat case) and hyperbolic 
trigonometric (bounded case) first order differential operators in the Hamiltonian; see e.g. 
[§] for the flat case. 

The paper is organized as follows. In Section 2 we give an abbreviated introduction of 
weighted Bergman spaces on bounded symmetric domains and fix some notation. In Sec- 
tion 3 we present the Segal-Bargmann transform and Berezin transform on real bounded 
symmetric domains, thus establish the abstract orthogonality relation for the spherical 
transforms of the Segal-Bargmann transforms of the L-invariant polynomials. We identify 
those Schmid components Vn which contain non-trivial L-invariant polynomials in Section 
4. In Section 5 we express the radial part of the Cayley-Capelli type operator in terms of the 
Dunkl operator. Their eigenspace decomposition is done in Sections 6 and 7. In Sections 
8 and 9 we study their Segal-Bargmann transforms and prove their orthogonality relation 
and find the expansion of the Bessel and spherical functions. We evaluate the constant in 
the Plancherel formula for the symmetric space H/ L in our settings in Appendix 1, and we 
list all the real forms H/L of a general Hermitian symmetric space in Appendix 2. 

Acknowledgement. I would like to thank Eric Opdam for providing me a copy of the 
Macdonald lecture notes [ |2~2~| ] and Charles Dunkl for sending a copy of his works. I thank 
Toshiyuki Kobayashi, Hjalmar Rosengren, Siddhartha Sahi and Harald Upmeier for some 
enlighting discussions. The hospitality of the Newton Institute, Cambridge, is also greatly 
acknowledged. 

For the reader's convenience we list the main notation used in this paper: 

• D = G/K a complex bounded symmetric domain of tube type and rank r' in a vector 
space V c , g = t + p the Cartan decomposition. 

• V a real form in Vc and r the conjugation of Vc = V + iV with respect to the real 
form V. 

• D = H/ L a real bounded symmetric domain in V, h = [+ q the Cartan decomposition 
of h, q is identified with V. 

• r: rank of the real bounded symmetric domain D, so that the rank r' of the complex 
domain D is r if D is of type D r , and it is 2r if D is of type C r . 

• ei, . . . , e r : a frame of the Jordan triple V. 

• a C q the maximal abelian subspace of q spanned by the vectors £j = £ ej , S the root 
system of f) with respect to a; 

• Pi, ... , P r E a* the dual basis of . . . , |£ r , identified also as linear functional on 
V. 
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• a the root multiplicity of and are independent of j, k and choice of the frame. 
Observe that dim Ajj = 1 and dim Bjj — t — 1; dim Bjk = dim A,^ = a for type C 
and D r (r > 3). 

• 7!, . . . , 7 r (if type D) or 7 1; 7^ . . . , 7 n 7^ (if type C) the Harish-Chandra strongly 
orthogonal roots; a' the non-compact root multiplicity of 7j ~ 7fc . 

• 6 = 6+ + 67 induced Cartan decomposition of t by r with = 1 = t(~)q 

• t = t~ + t+ the induced decomposition of a Cartan subalgebra t of t. 

• c„ the normalization constant for the Berezin transform, c' v the normalization constant 
for the weighted Bergman measure, C$ the one for the Plancherel formula on D = 
H/L, C\ the one for the integral of L-invariant functions on V in terms of polar 
coordinates on 0. 



2. Complex bounded symmetric domains 

We recall very briefly in this and next sections some preliminary results on bounded 
symmetric domains and fix notation; see [ p] ] and [0] and reference therein. 

Let D = G/K be an irreducible bounded symmetric domain of tube type in a d- 
dimensional complex vector space Vc = C d of rank r'. (The symbol r will be reserved 
for the rank of the real bounded symmetric domain D in Section 3.) Let g = 6 + p be the 
Cartan decomposition and g c = p + + t c + p~ be the Harish-Chandra decomposition of its 
complexification. The space Vc = p + has then a Jordan triple structure so that the subspace 
p is of the form 

(2.1) P = {v-Q(z)v,veV c }, 

when the elements are realized as holomorphic vector fields, where Q(z) : Vc 1— ► Vc is qua- 
dratic in z. We denote {xyz} the Jordan triple product {xyz} = (Q(x+z)—Q(x)—Q(z))y. 
We fix a ^-invariant Hermitian inner product (•, ■) on Vc so that a minimal tripotent has 
norm 1. We let dm(z) be the corresponding Lebesgue measure. The Bergman reproducing 
kernel up to a positive constant is of the form h(z, w)~ p where p is the genus of D, defined 
by V — fr (f° r turj e domains) and h(z, w) is an irreducible polynomial holomorphic in z 
and anti-holomorphic in w. 

Let v > p — 1 and consider the probability measure dii v (z) = c' v h(z, z) u ~ p dm(z) where 
c' u is the normalization constant, and the corresponding weighted Bergman space H. v = 
7i v (p>) of holomorphic functions / so that 

11/112= / |/Wl a d^)<oo. 
It has reproducing kernel h{z, w)~" '. The group G acts unitarily on H u via the following 

(2.2) ir v f(z) = J g -,(z)7f(g- 1 z), 
and it forms a unitary projective representation of G. 
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Let T v = J- v (Vc), for v > 0, be the Fock space of entire function on Vc with the norm 
defined by 



7T 



'V c 

Thus it has reproducing kernel e u ^ z ' w \ When v = 1 we write for simplicity T = T\. The 
norm in JF can alternatively defined by 



(2.3) (f,9h„ = v- des(p) f(d z )g*(z) 



2=0 



for polynomials / and g, where g* is obtained from g by taking formerly the complex 
conjugate of the coefficients of the monomials (in terms of an orthonormal basis in Vc). 
The first part of the next result is due to Hua fll6| ] for classical domains and Schmid [ p0| ] 



for general domains, the second and third part is due to Faraut and Koranyi [[7p, Theorem 
3.8; this result will be of fundamental importance for our work here. Fix a Cartan subalge- 
bra of t and let 71 > • ■ - > 7,./ be the Harish-Chandra strongly orthogonal roots. Let a' be 
the root multiplicity of 7j ~ 7fc in p + . 

Theorem 2.1. The space V ofholomorphic polynomials on Vc decomposes into irreducible 
subspaces under Ad(K), with multiplicity one as: 

(2.4) V = J2Vn- 

n>0 

Each V R is of lowest weight — n = — (71171 + ■ • • + n r ^ r ) with n\ > ■ ■ ■ > n r > 0. The 
quotient of the norms of a polynomial f G in the Fock space T and in the weighted 
Bergman space H u is given by 

where 

r' , r "i , 

Ma = U( v = II 11^ - - !) + * - !)■ 

j=l 3=1 k=l 

is the generalized Pochhammer symbol. 

Let Kn be the reproducing kernel of V R with in the the Fock space T v for v = 1. Then 
as a consequence we have, writing |n| = ri\ H 1- n r >, 



(2.5) e >'M = ^ u Nif ji ( Z) 

n 

and 

(2.6) &(*,™)~ V = X>)ntfn( 



re 



2, w . 
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3. Berezin and Bargmann transform on real bounded symmetric 

DOMAINS 

Let V be a real form of Vc and r the complex conjugation with respect to V. Suppose 
r(D) = D, namely, r fixes the bounded symmetric domain. Then the real form D — D n V 
is called a real bounded symmetric domain. In this case the triple product D(x,y)z = 
{xyz} restricted on V defines also a triple product on V. 



A complete list of real bounded symmetric domains D is given in pip. As a Riemannian 
symmetric space, D = H/ L, where H is the connected component of the subgroup of G 
of biholomorphic transformations of D which keep D invariant. The coset space G/H is 
called a causal symmetric space, a complete list of the pairs (G, H) can be found in e.g. 



Q25Q and QT3|], see also Appendix 2. 
The iJ-invariant measure on D is 

(3.1) h(z, z)~2dm(z), 

and H acts unitarily on L 2 (D, hr^dm) via change of variables, 

(3.2) Mg)f(x) = fig' 1 *), g^H. 

We describe briefly some algebraic and geometric structures of the domain D. 
Let h = q © I be the Cartan decomposition of the Lie algebra f) of H. Similar to ( PTTj ) we 
have 

(3.3) q = {v-Q(z)v,v E V}. 

We thus identify q with the underlying space V via the mapping v i— > £ v (z). 

Let r the rank of D = H/L, so that the rank r' of D will be r or 2r; see below. Let 
{ej,j = 1, ... ,r} be a frame of minimal tripotents in V. The elements £j = £ ej = 
ej — Q(z)ej, j = 1, . . . , r, span a maximal subspace a of q of dimension r. Let {/3j,j = 
1, . . . , r} in a* be the dual basis of 

where 6j k is the Kronecker symbol. Then the root system £(jj, a) is of types A r , C r or D r . 
Type A r corresponds to the case that V is a formal real Jordan algebra; we will be only 
concerned with type C r and type D r {r > 3): 

£(a,a) = {±&,^|^}; 

and respectively 

S (fl ; a) = {^— }. 



We let, as in [|39|], a be the root multiplicity of l3]± ^ k and i — 1 that of /3j. Arrange an 
ordering of the roots so that 

Pi<fa<~-<P r . 
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The ranks r' and r and the multiplicities a', a and 6—1 are related to each other: 

(3.4) r ' = r J a' = 2a 
if it is type D r (r > 3), and 

(3.5) r' = 2r,a = 2a', L = 2 + a'; 

if it is type C r for r > 2; a' = and i = 2 + a' if r = 1. Those relation can easily be 
obtained by calculations of the dimensions of the root spaces or by a case by case check of 
the table in the Appendix 2. 

We note that with the above normalization of the inner product on Vc a minimal tripotent 
of V has norm v2 if the root system is of Type C, otherwise it is of norm 1; see below. 
(The inner product in [ |39| ] is normalized so that a minimal tripotent of V always has norm 
1.) 

To consider the branching law of (H u , k v ) of G under H we let R be the restriction map 
<$8#)R:H V ^C°°(D) by 

(3.6) Rf(x) = f(x)h(x,x)%, xeD. 

Then R is an //-intertwining map, as one can easily checks from the transformation prop- 
erties of h(x, x). Consider its formal conjugate operator R* from L 2 (D, dfio) to H v and 
form the operator R*R on L 2 (D, uj). It is 

(3.7) RR* = —B u , 
where 

h(z, ) 2h(w, w)% dm(w) 



(3.8) BJ(z) = c v \ f(w)- 

Jd 



h(z,w) v h{w,w)z' 
is the (normalized) Berezin transform and the constant c u is such that B v l = 1. The 
constant c u is evaluated in [|3"Q[]. Let 

(3.9) R=\R\U 

be the polar decomposition of R. Thus \R\ 2 = RR*. 

It is proved in p^ ] that when v > p — 1 for type C and suppose v > | — 1 the Berezin 
transform RR* is bounded, and that the multiplier h% is in L l {D, h~^dm(z)). Since for 
those values of v the space Ji v contains all polynomials, thus the range of R contains 
the functions of the form h(z, z)*p(z, z) where p(z, z) are polynomials of z and it clearly 
forms a dense subspace in L 2 (D, h~^dm{z)). This proves 

Proposition 3.1. Suppose v > p — 1 for type C and suppose v > | — 1 for type D. The 

2/ 



operator U is unitary and intertwines H -actions ( |2.2| ) on 7i v and ( p.2| ) onto L ' (D). 



Definition 3.2. The unitary operator U is called a (generalized) Segal-Bargmann trans- 
form. 
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Let <px be the spherical function on D = H/L. Let 

dm(z) 



/(A) = / f(z)Mz) 



D 



h(z, z) 2 

be the spherical transform on D, where / is a L-invariant C°° -function on D with compact 
support. It is well known ([jTlp, Chapter IV) that the map extends to a unitary operator from 
L 2 (D) L onto L 2 (a*, C \c(X) \~ 2 d\) w of VT-invariant functions in the L 2 space on a*. Here 
c(A) is the Harish-Chandra c-function, with the same normalization as in [|TT)], Chapter IV, 
dX is the regularly normalized measure and the constant Co can be evaluated by using the 
evaluation of the constant c„ in and the evaluation of the Selberg integral, see Appendix 
1. 

Corollary 3.3. The map f i— > Uf is a unitary operator from onto L 2 (a* , C a | c( A) | ~ 2 dX) 

In particular if we find a canonical orthogonal basis TC^, their image under the above 
unitary operator then gives an orthogonal basis L 2 (a*, C \c(X) \~ 2 dX) w . 

4. Existence of the L-invariant holomorphic polynomials 
We identify now those polynomial spaces Vn that contain L-invariant vectors. 
Lemma 4.1. In the decomposition (\2.4\) the component ("Pn) L 7^ if and only if 



w 



(4.1) n= (mi,m 1 ,m 2 ,m 2 , . . . ,m r ,m r ) = ^rrij (723-1 + 72 j) 

3=1 

ifH is type C r , and 

r 

(4.2) n = (2mi, 2m 2 , . . . , 2m r ) + m(l, 1, ... 1) = J^m,- + m)7j- 

//"S is of type D r , where in all cases rrij and m are nonnegative integers and mi > m 2 > 
■ • ■ > m r > 0. 

Proof. The involution r on Vc induced an involution on t and its fixed point set is [, thus 
(t, V) is a symmetric pair. Let t = I + t~ be the corresponding Cartan decomposition. Let 
t~ C t~ be a maximal abelian subspace of t~, and t = 6+ + 6^: a Cartan subalgebra of 
t with t+ C [. We apply the Cartan-Helgason theorem [|T2|], Chapter V, Theorem 4.1, to 
identify the L-spherical representations, which asserts in our case that the representation 
Vn contains a L-fixed vector if and only if 

(4.3) n| ( e+)c = 
and 
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for all roots a in the root system E(t c , (t 7 7) c ) of 6 C with respect to t~, in which case is 
one-dimensional. Here N is the set of nonnegative integers. 

Consider the type D r first. By (3.1) the frame {ei, e 2 , . . . , e r } of minimal tripotents in 
V is also a frame in Vc- Recall the involution r induced by the complex conjugation on Vc 
with respect to V. We have t(£ 6j ) = £ e . r (&<,.) = — £ e .. Thus 

T{iD{e j ,e j )) = -iD{e j ,e j ), 

since 

iD{e jl e j ) = -[^,6eJ, 

so that iD(ej, e,-) E t~, j = 1, . . . , r, and they span an r-dimensional abelian subspace 

i(MD(ei, ei)H \-RD(e\, ex)) of t~, whereas the dimension of t~ is dim(t~) = rank(i^/L). 

We claim that rank(if/L) > r. Indeed the symmetric pair (t, I) is (s(u(r) © u(r)), so(r) © 
so(r)), (u(2r),so(2r)) or (so(10) + R,sp(4)) (with r = 3), the rank of the first pair being 
2r — 1 > r, the second 2r > r and the third 7 > 3 (see []TT|], Table V, p. 518). This means 

that the subspace i(RD(ei, ei) H h RD(ei, ei)) is a nontrivial subspace of So the 

vanishing condition ( |4.3[ ) is already satisfied since n is vanishing on the orthogonal com- 
plement of z(RD(ei, ei) + • ■ -+RD(ei, ei)) in t. To check the second condition, recall (see 
©, formulas (16) and (17)) that all roots of in £(£ c , (t;) c ) are of the form ^^ + a, with 
a orthogonal to 7j ~ 7fc and if nonzero, ||a|| 2 = |[ 7j ~ 7fc || 2 . Notice that since 7j ~ 7fc , viewed as 
linear functional on t~, only span a r — 1-dimensional subspace, thus there exist roots in 
£(E C , (t~) c ) of the form 7j ~ 7fc + a with nonzero a for all j,k = 1, . . . ,r, j ^ k. Thus by 
the Cartan-Helgason theorem, the polynomial representation n is L-spherical if and only if 



for all j < k, and 



n 7 - - G Z> , 



l lj-lk 7j~7fc \ 
\ 2 ' 2 / 

when a = 0. The first condition clearly implies the second and it is just our stated condition. 

Now consider type C r . There are only three cases and we study them case by case. 
Consider the case (I, I) = (s(u(2r) © u(2r)),sp(r) © sp(r)). The highest weight n = 
(miSi+m 2 e2+- ■ -+m 2r £: 2r ,)®(mi£:i+m 2 £: 2 +- • ■+m 2r £2r)* where Ej is the dual of diagonal 
matrix with jth entry being 1 and rest 0, in the standard matrix representation of u(2r), and 
(mxSx + m 2 e 2 + ■ • • + m2 r €2r)* is the contra-gradient representation. The representation m 

has a [ = sp(r) © sp(r)-fixed vector if and only if the representation mx£x + "12^2 H h 

m 2r .e 2r . has a sp(r)-fixed vector, and the later happens precisely when mj = m 2 ,m 3 = 
777,4, • • • , rri2r-x = m2r, namely our condition, again by the Cartan-Helgason theorem, since 
(su(2r),sp(r)) is a symmetric pair. Now let (t, 1) = (u(2r), sp(r)). The Harish-Chandra 
roots are jj = 2£ 27 -_ 1) 7^ = 2e 2 j- Our result immediately follows from the above argument. 
Finally consider (t, 1) = (so (2) © so(p),so(p)). The highest weight n = 7712(71 + 72) + 
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(mi — m 2 ) (71), and as representations n = m 2 (7i + 72) <8> (mi — m 2 )7i with m 2 (7i + 72) 
being the trivial representation of so(p) and (mi — m 2 )7i the representation of so(p) on the 
spherical harmonics of degree mi — m 2 , the later has an so (p) -invariant vector if and only 
if mi — m 2 is even. This completes the proof. □ 

Remark 4.2. In [|T9p Kramer has given a classification of all spherical connected sub- 
groups L of a simple compact Lie group K (Tabelle 1, loc. cit) and listed all the fun- 
damental spherical highest weights. Our result can also be deduced from that list. One 
may also prove the above result somewhat by slightly general argument by following the 
classification as in p5| ] and Moreover the above result for type C holds also for type 



BC and that for type D holds also for type B (with m — 0). 

Let A (z) be the determinant polynomial of the Jordan triple Vc and A (d) the corre- 
sponding differential operator which we call the Cayley type operator. The next result 
gives the eigevalue of the operator A(z)~ a A(d)A(z) a+1 , sometimes also named as the 



Cayley-Capelli operator, under the Schmid decomposition (|2.4D, and its follows easily by 
using Theorem 2.1; see e.g. [Q, [[151] . 



Lemma 4.3. Let r' be the rank of 3. The differential operators A(z)~ a A(d) A(z) a+1 for 
r' different nonnegative integers a form a system of generators of K -invariant differential 
operators onV(Vc). A holomorphic polynomial f € V(Vc) is in the space V R if and only 
if it is a solution of the system of differential equations 

r' , 

(4.5) A(z)- a A(d) A(zr +1 p(z) = J](-(r' -j) + l + a + nj )p(z), 

for r' different nonnegative integers a. 

In the next sections we will find the L-invariant polynomials in 7\- 

5. Determination of the L-invariant holomorphic polynomials: Some 

general results 

We consider the Chevalley restriction map Res from V(Vc) L onto the space V(a) w of 
jy-invariant polynomials on a. By using the Dunkl operator we define an inner product 
on V(a) w and we prove that the restriction map Res is an unitary map from V(Vc) L with 
the Fock space norm onto V(a) w . This reduces the problem of finding and calculating of 
L-invariant polynomials to the corresponding one of VT-invariant polynomials on a. 

Some consideration that follows will be true for some general root system, we shall, 
however, only consider the root systems S as in Section 3. 

We fix a frame {ej,j = 1, . . . , r} of minimal tripotent in V, enumerated so that ej £ 
V C V c = p + is a root vector of the Harish-Chandra orthogonal root jj, j = 1, ... r, if E 
is of type C, and ej is a sum of two root vectors with roots 72^-1 and 7 2j for type D. 
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For any root a G £ let r a e be the reflection defined by a. We recall the Dunkl 
operator M\, 



2 a(x) 
acting on polynomials f(x) on a, where the r a acts /(x) via 

(r a f)(x) = fir^x). 

The operators Dj,j = 1, . . . , r are pairwise commuting, and thus define an isomorphism 
between the ring of polynomials on a and the ring of difference-differential operators gen- 
erated by them. We can thus define, for any polynomial x = X\e\ + ■ ■ ■ + x r e r — * f(x) = 
f(xi, . . . , x r ) the operator f(D) by assigning Dj to the polynomials Xj, j = 1, . . . , r. 

Definition 5.1. (Dunkl []5]]) Let E be the root system of type C or type D. The S-inner 
product on (V(a)) w is defined by 

(f,gh = f(D x )g*(x)\ x=Q , 

if E is of type D, and 

(f,gh = f(~D x )g*(x)\ x=0 

if E is of type C. 

The discrepancy for type C here is due to the unmatched norms of minimal tripotents in 
Vc and V. 

Proposition 5.2. The restriction map Res is an isometric mapping from (V(Vc) L , \\ ■ 
onto (V(a) w , || ■ || s ). 

Proof. We consider the operators 

(5-1) E= l -(z 2 l + --- + z 2 d ) ) 

(5-2) F = ~(a? + ... + 3j), 

and 

(5.3) H = { Zl d x + ■■■ + z d d d ) + - 

acting on the space V(V<c) L . Then it is clear that they form the Lie algebra s[(2, C): 

[E,F] = H, [H, E] = 2E, [H, F] = 2F. 
Let us, following Heckman [|T0|], define similarly 

(5.4) Eo = ^(xi + --- + x 2 r ) } 



(5.5) F = -±(Dl + ..- + D, 
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,2\ 



for type D, and 

(5.6) E = x\ + • ■ ■ + x 2 r , 

(5.7) F = -^(D 2 1 + .-- + D 2 rJ , 

for type C (again due to the ill-matching of the norms of minimal tripotents in Vc and V), 
and 

(5.8) H = (x 1 d 1 ^ h x r d d ) + -(r + J^m a ), 

aeR 

for all types. They form a copy of the Lie algebra st(2, C). We claim that 

Res E = E , Res F = F , Res H = H . 

The second is trivial, the third follows since that the dimension d = dim c Vc = dim R q = 
§ + § J2 a &R m a- The first is just the formula for the radial part of the Laplace operator on 
q; see [[12|], Proposition 3.13 for the formula and 0] (or [JT0|] ) for the calculation of F . It 
is proved in [jKj Proposition 3.4 that for any polynomial p of degree m on a, viewed as 
multiplication operator on V(a), namely in Aut(V(a)), 

(5.9) p(D) = (-ir^ a d(F r( P ), 

where the Lie algebra sl(2, C) is acting on Aut(V(a)) via the adjoint action. The essentially 
same (even easier) calculation shows that, for polynomial P on V(V C ) we have 

(5.10) P(d) = (-lr^r &d{F) rn {P) 

ml 

For any P,Q e V(V C ) L , let p = Res P, q = Res Q;ifxE a, 
P(d)Q*(x) = (ResP{d)Q*)(x) 

Res((-ir^- a d(Fr(P)Q*)) (x) 
ml J 

(-l) m ^ T ad(F ) m (ResP)ResQ* ) (x) 
ml J 

— p(9)q*(x) 

and 

(P,Q)r = P(d)Q*(0)=p(d)q*(0) = (p,qh = (ResP,Resg) E , 
completing the proof. □ 

It is noted in [ JT0| ] that the above idea of reducing computation to second order operators 
goes back to Harish-Chandra. 

Remark 5.3. The above result clarifies the significance of the Dunkl operator and the inner 
product (•, -)s- Moreover it gives an isometric version of the Chevalley restriction theorem 
Chapter 2, Corollary 5.12). It seems to the author that this was not been known 
before. 



14 GENKAI ZHANG 

As a consequence we get 
Corollary 5.4. The Cayley operator Res A(<9) on L-invariant polynomials is given by 



for type D and 



for type C. 



(ResA(d))f(x) — Y^Djf(x) 
3=1 



(ResA(d))f(x)=2~ 2r l[D 2 3 f(x) 

3=1 



Proof. We consider only the type D r , the type C r -case can be proved similarly. Recall that 
the determinant function A on Vc is of degree r and when restricted on a it is 

r r 8- 

ResA(x) = A(xiei H \-x r e r ) = T\xj = ( JJ — )(x). 

i=i j=i 

We calculate now the adjoint operator of A and Res A. The adjoint operator of multiplica- 
tion by A(z) on V(Vc) with respect to the Fock norm is A(d), A* = A(d) and respectively 
the multiplication by n^=i 2 1S (Ilj=i 2 )* = T[j=i Dj with respect to the E-norm by the 
definition of the inner product. Thus for P,QE V(Vc) l we have, by the preceding propo- 
sition, 

(Res(A(0)P),ResQ) fl = (A(d)P,Q)r = (P, AQ) r 

= (ResP, Res AQ) E = (ResP, (fj ^) Res AQ)e 

3=1 

= ((Q J Y ^s P, Res Q) E = ((fjDj) ResP, Res Q) s 
3=1 j=i 
proving the result. □ 

6. L-INVARIANT HOLOMORPHIC POLYNOMIALS: TYPE C r 

In this section we find the L-invariant polynomials in P„ and calculate their norm in the 
Fock space. We will express them in terms of the Jack symmetric polynomials. For that 
purpose we recall some basic facts. 

Let 

(6.1) D] = Dt = 9j + ^-±- {1 - Sij) 



be the Dunkl operator ([gp, [|K)|]) acting on functions on the r-dimensional vector space ' 
where the superscript indicates that the underlying root system is of type A. Let 

(6-2) Uj = Uf = D jVj -^J2 
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be the Cherednik operator for type A ([Q] and [0]). Then U^, j = 1, . . . , r are commuting 
operators. The relation between llf and the Cayley type operator n*=i Dj ]X=i Vj 1S 
the following. (It can be viewed as a symmetric invariant analogue of the Cayley-Capelli 
identity, which express the product A(d)A(z) as another determinant.) 

Lemma 6.1. The following identity hold 

(n^)(n%)=n^ 

j=l j=l j=i 

Proof. The proof relies on the commutation relations -DjSjj = s i3 D 3 - and [Di, xf\ = — f s^. 
From this we deduce that 

r r 

l[D J l[y J = D 1 ...D r _ 1 D r ym ■■■Vt 

3=1 3=1 

r—l r—1 

= D x . ..B r _ x y x D r y 2 ...y r - -(JJ Dj)(Y]_yj)s lr ; 
repeating the argument (moving D r until it reaches y r ) we get 

r r 

(6.3) l[D j ]ly j = D 1 ...D r -\D\V2 ■ ■ ■ y r -iU r . 

3=1 3=1 

Performing the above computation with D r _i and so on proves the formula. □ 
Essentially the same computation gives 

r r r r 

(6-4) (Uuj) Yl P ^y = HP, + a). 

3=1 3=1 3=1 3=1 

Remark 6.2. The Cherednik operators Uf' are introduced before (see for root systems 
of type B and reference therein) in order to study the non- symmetric Jack polynomials. 
The above lemma shows that it can also be obtained, though less systematically, in trying 
to write the Cayley-type operator dlj-i -^jXIl^i x j) as a product of r operators. 

The Jack symmetric polynomial are then a sum of joint (nonsymmetric) eigenfunctions 
of Uj. Let f2 m be the normalized Jack symmetric polynomial, so that it is a symmetric 
eigenpolynomial of the operators nj=i(^j ' + a ) 

r r 

(6.5) \[{Uf + a)O m = (n(^( r ~k) + l + a + m k ))tt m ( yi , ...,y r ) 

3=1 3=1 

for all nonnegative integers a, normalized so that 

n m (i,...,i) = i. 
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Let Dj be the Dunkl operator on a = Rei + . . . Me r for the root system E of type C, 



d 3 = d 3 + - 8j ) + 1 E(r-^:( 1 " *«) + -^-(i - 

where <7j, Sy and <7y are the reflections in Weyl group corresponding to the roots 7,-, |(7; 
7j) and respectively + 7,-). 

Recall that the Hermitian form on V is normalized so that tj has norm a/2. 

Lemma 6.3. The restriction of the operator Res(A(d x ) A(x)) on a w gzven /Vy 
Res(A(cyA(x)) 

= 2 ~ 2r n ^? n *? = 2 - 2r n ik d ^ - \ ek- + ^ ri ^ 

3=1 3=1 3=1 3=1 " i<j j=l 

= 2~ 2r f[ (d jXj - a ~ + Sij ) I)*,) j n - 1 ek- + s ^ 

j=l \ i<j J j=l \ i<j 

Proof. The first equality follows by Corollary 5.4, for the restriction of A(x) on a is 
YYj=i x "j- The rest of the proof is similar to that of Lemma 6.1. We have the product 
formula 

r r r 

j=l j=l j=l i<j 

and shift formula 

r r 

(II D k)( D 3 X 3 - 2 + S ^ = ~ 2 + ^ + 1 - ~ iKXlI Dk ^ 

k=l i<j i<j k=l 

which can be obtained by repeatedly using the commutation relation 

[ D 3> X k] = \( a 3k ~ S 3k), U ^ k), 



[Dj, Xj ] = 1 + (t - l)Sj + I ^((Tij + Sij), 

i+3 



and 



a 3 D 3 = 



^3^3"> ^ij-^j ^i^ij-) ^ij^j -Di&ij' 
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Consequently 



3=1 3=1 



X 3' 



n /; nn /; nil' 

3=1 3=1 3=1 3=1 



r r r 

= (II A) life - |X>« + ^))(II^) 

3=1 3=1 »<3 3=1 

r r r 

= life - fx>« + s ^ + 1 - - ij^cn^di^) 

3=1 i<3 3=1 3=1 

r r 

= Y[(D jXj - - J2( a H + s v) + 1 - ( L - l ) a i) \\i D i x 3 ~ 2 + s «)) 

3=1 i<3 3=1 i<3 

□ 

Lemma 6.4. 77ze operator ~Res(A(d x ) A(x)), when acting on W -invariant polynomials of 
the form f{x\ 1 . . . , x 2 ) with f(yi, . . . , y r ) being a symmetric polynomials in r variables, is 
given in terms of the coordinates yj = xj by 

T T 

Res(A(d x )A(x))f(x) = \[{Uf + -{t -i)--)f[uf 

3=1 3=1 

where Uf is the Cherednik operator ( |5.2| ) of type A acting on functions ofy\,...,y r . 

Proof. We use the previous lemma and consider the operator DjXj — | J2i<j( a ij + s «i) 
acting on the functions of the form f(yi,---,y r ) — f{x\- l --- ) x 2 r .) ; this operator maps / 
into functions of the same form, and in terms of the variables yj = x?it is 

(6.6) 2(Df - 2 £ s„ + t^L - i) = 2(Uf + iZl - i) 

«<3 

by direct calculation, proving our Lemma. □ 

The above lemma can also be proved by using [g], Proposition 5.2, where the operator 
(rij=i Dj)(\Xj=i Xj) is expressed in terms of the Cherednik operators. (Note that our op- 
erator Uf differs by a constant with the operator Uaj there.) Together with ( |6.5[ ) it implies 
then 

Lemma 6.5. The Jack symmetric polynomials Q m (xl, . . . , x"l) are eigenfunction of the op- 
erator Res(A _a A(<9)A 1+a ) with eigenvalue 

r 

(6-7) l[(^(r + ^ + mj + or)(|(r - j) + rrij + a). 

3=1 
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Proposition 6.6. For each m = (mi, . . . , m r ) there exists a unique polynomial p R in the 
space with n given by m as in such that 

(6.8) Respn(xi, ...,x r ) = Q m (xl, ...,x 2 T ). 

Proof. The existence of such polynomial p„ in V( V<c) L is by the Chevalley restriction the- 
orem. Lemma 4.3 implies that p„ is in the space Vn, with n as given, if and only if it is an 
eigenfunction of the operator A~ a A(<9) A 1+a with eigenvalue 



2r 

,a' 



Y[(-(2r-j) + l + nj + a) 

r 

JJ (|(2r-2z + l) + l + n 2i _i + a) JJ (|(2r - 2i) + 1 + n 2i + a ) 

j=2i-l,i=l j=2i,i=l 
1 r 

J[(o'(r - % + -) + 1 + m» + a) JJ(o'(r - i) + 1 + m; + a). 



2 

i=l i=l 



Using ( P3] ) we see that the factors in the product ( ]5T7| ) are 

^(r — j) + ^ + rrij + a — a'(r - j) + 1 + ^ + + a = a'(r - j + -) + 1 + m,- + a 
and 

^(r - j) + 1 + m,,- + a = a'(r - j) + 1 + m.,- + a, 
so that ( |677| ) coincides with the previous formula. □ 

We calculate now the Fock space norm of Pn by a direct calculation using certain recur- 
rence formula of Macdonald. 



acting on W -invariant polynomials has the following form, 



Lemma 6.7. In terms of the coordinates yj = Xj,j = 1, . . . ,r, the operator F when 



(6-9) F = -{-2F* + (|(r - 1) + \ + ^) £ ^ 

where Fq is the corresponding operator for the root system of type A, 
(6-10) itf = - l -Ct{Dff) = -i (E%(^) 2 + a E 



Vi_ d {v) 



1 Vi ~ Vj 



With some abuse of notation we denote f2m the function = fim(^i, ■ ■ ■ ,x^). 

To state our next result we let ( m ) be the generalized binomial coefficients [ p0[ ] and let 
m respectively m j stand for the signature m - = (mi, ■ ■ • , m r ) — (0, . . . , 0, 1, 0, . . . , 0) 
and = (mi, ■ • • , m r ) + (0, . . . , 0, 1, 0, . . . , 0) (with 1 in the j'th position). The binomial 
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coefficients ( — ,) is then (loc. cit., Section 14) 

m\ a / . NN -j-j- m , ■ — ///, - \ ( / — j - I ) 



(m\ . a , xx -n- 

- J = to + -(r - 
3' i=^i 



Lemma 6.8. 77ze operators E Q and F have following upper respectively lower triangular 

(x) 

form when acting on the polynomials f2m 

r 

(6-12) EoOg = ^cJj)QM, 

3=1 

(6.13) 



F nW = -i g (4K- - 1 - |(j - 1)) + 2a(r - 1) + 2(, - 1) + 2) ( ^ J 

n mj - mj + f(l + i-j) 



Proof. This follows from our formula ( J6.9D and the result of Macdonold [[22]], Section D. 
Following temporarily the notation there, let 

and 



r 



(2/). 



it is proved by Macdonald that 



□lOn = EK - 1 - SO' - !)) f m )"r 

7 =1 ^ V— j/ 



r 

and 



r / \ 

- Vm 7 7 - J 



3=1 3=1 

which then imply our result. □ 
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Theorem 6.9. The norm square ||pn||^- of the L-invariant polynomial pn, with n deter- 
mined by m as in ( ft. is given by 

l<i<j<r \ < 2^ J II j=i j=i 

-r-r T(m, + l-m j + Z(j 

x iii<r T ^ - m > + 2U + 1 - *))( m * - m J + 1U - *)) 

Proof. For a fixed j write m' = = (mi, . . . , m^-i, — 1, . . . , m r ) and let n' 
be the corresponding n. p R and p n ' are orthogonal in JF as they are in the different Schmid 
components, so are the polynomials Qm and fl^ with the S-inner product by Proposition 
5.2. Now, on the space of all VT-invariant polynomials, the adjoint (E )* = —F with 
respect to the S-inner product, from this we obtain 

this gives 

\\n [ M_(m\rn J -± + ^ + l(r-j) 



\\n^,\\l \m'J <w(j) 

This recursion formula, together with the fact that (po,Po)r — (^o,^o)e = 1 uniquely 
determine the norm. Carrying out the calculation gives our result. □ 

7. L-INVARIANT HOLOMORPHIC POLYNOMIALS: TYPE D r (r > 3) 

The Weyl group W in this case consists of the signed permutation of vectors (x±, . . . , x r ) 
keeping the product x\. . .x r invariant. Thus any ^-invariant polynomial is of the form 
[x\ . . . x r ) m f(x\, . . . , xf.), where / is a symmetric polynomial in r variables. The Dunkl 
operators are 

D f = d i + % E(^— t 1 - s ^ + -^—o- - *■«))■ 

J / y . Y' ■ T* ■ T' ■ 

We are interested in the operator (IX/=i Dj) {Y[ T j=i x j)- Similar to the proof of Lemma 6.1 
we have 

r r— 1 

(7.1) D r (JJa?i) = (JJa;i)(A^r - -^(s ir + (7i r )), 

i=l i=l i<r 

and generally 

(7.2) /; ^n r) ( n 

i<3 i<J-l i<j 



We define therefore Cherednik operator Dj, j = 1, . . . , r for type D, 

Uf = D jXj - - J2( S iJ + a ij)- 



2 

Kj 
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Thus we have 

r r r 

(7.3) (n^xrN = n^ 

j=i j=i j=i 

Moreover, it is easy to prove that 

r r 

(7-4) U°([[x j ) = ([[x J ){Uj> + l), 

3=1 3=1 

Therefore we have 

Lemma 7.1. We have the following formula 

3=1 3=1 3=1 

where (T) m = T(T + 1) . . . (T + m — 1) is the Pochhammer symbol of any operator T. 

Consider the Cherednik operator XJf acting on even functions f(xf, . . . , x^). Performing 
the change of variables y 3 - = x|, we have 

1 



(7.5) Uf = 2{Uf - 



D 



where U A is the operator (6.2). This proves consequently that the Cherednik operators U, 
are also commuting operators when acting on even functions. 

The next result follows then immediately from Corollary 5.4, the formulas (7.3) and 
(7.4). 

Lemma 7.2. The operator Kes(A(d x ) A(x)), when acting on even polynomials of the form 
f(xf, . . . , x 2 r ), is, after the change of variables y 3 = x 2 -, given by 

Res(A(x)- a A(d x )A(x) 1+a ) = T f[(Uf -\ + \) 

3=1 

where XJf = Dj Vj — f J2i<j a ij zs tne Cherednik operator in variables y 3 - 's. 

Proposition 7.3. The polynomials (YYj=i x j) m Qm(. x h ■ ■ ■ > x r) are eigenfunction of the op- 
erator Res(A(x)~ Q A((9 x ) A(x) 1+a ) with eigenvalue 

r 

JT(a(r -k) + 2 + a + m + 2mf). 

3=1 

Moreover, it is the restriction of a unique L-invariant polynomials p n in the space V n , 



(7.6) RespnOi, ...,x r ) = (JJxj)" 1 ^^, . . . 

3=1 



x 2 ) 



with n given by m as in ( \f.2\ ). 
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Proof. The operator Res(A(x)~ a A((9 ;c )A(x) 1+a ) has the form 



r r r 



(n%)" a (n^)(n^) i+a 

j=i j=i j=i 

by Corollary 5.4. We calculate its action on (115=1 x j) m ^m( x i, • • • > x 1)- ^ ls 



r 



3=1 V 3=1 3=1 3=1 / 

Applying Lemma 7.2 for the operator in the parenthesises and using (6.4) we see that our 
polynomials is indeed an eigenfunction with eigenvalue 

„ r TT,a, ..lam . 

2 11^2 (r ~ fc) + 2 + 2 + 2" + mi) 

i=i 

as claimed. The rest of the proof is similar to that of Proposition 6.6. □ 
Theorem 7.4. The norm square ||Pn||^ of the L-invariant polynomial p^ is given by 

n r( m^'^-ffl' )) 22 "" i+ - + '"'' n<5 + - ^ n<i + §<•• - 

n\a 1 -j-j- r^ + l-m^ + Kj-l-z)) 

^ + 1 + 2 j - i n^rK-m, + |(i + l-i))K-m i + |(i-z)) 

Proof. Proposition 5.2 implies that 



\\ P ^ = ii Res^m = njiDi . . . ,D 2 r )(n Ar(n^) m %^. • • • 

i=l i=i 

However, by Lemma 7. 1 

j=i j=i j=i j=i 

which has Vl^xf, . . . , xf) as an eigenfunction with eigenvalue 



■> X r)\ x= Q- 



2 / m > 



by Q. Thus 

r 

— TT/^ W /; , >",; I lir- -in \ J--' J J--1U\' | ,r:li 



, 2 v ^ 3 2 , 

J'=l 



Ibnll^ = II(^ r - i) + 1 + ™J - ^m^W^?, • • • , iftOnfc?, • • • , ^ 



'2 
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The inner product above can be calculated by the same recursion formula as for type C. 
Indeed the Laplace operator F Q in this case, after changing of variables is of the form 



-2itf + (\(r - 



i=i / 



which is of the same form as ( |6.9t ) except the term L -^- is missing and that the coefficient 
— 1 in front is replace by —2. □ 



8. Bargmann transform of L-in variant polynomials: Flat case 

Consider again the restriction map 

(8.1) R = R V : F U (V C ) h- C"(V), Rf(x) = f(x)e-^ 2 

It defines a bounded operator and the isometric part U in the polar decomposition R — \R\U 
of R is a unitary operator, and is the Segal-Bargmann transform. The Berezin transform in 
this case is 



\R\ 2 f(x)=RR*f(x) = [ e-W'-^f{y)dy 

Jv 



Recall our identification of V with q. In this identification a = Rei + ■ ■ • + IRe r is a 
subspace of V. 

Define the Bessel function on V by 

(8.2) J x (x) = [ e- imx)) dl 



J L 

for A e a* where we extend the linear functional A on a to V via the orthogonal projection 
onto a and where dl is the normalized Haar measure on L, so that J\(0) = 1. 

Definition 8.1. The Hermite polynomial Cn,^(A) related to the root system E of type C or 
D on a is defined by the Rodrigues type formula: 

C a ,,(A) = J^Pb(9 x ) (e^ 2 A(x)) \ x=Q . 

for those n as determined in (|4~2"1). Here Pn(d x ) is the differential operator on V obtained 
from Pn(x) by the same convention as in (p3|). 



Conceptually it is better to write this as 

Cn,,(A) = ^—p^iR-'j^O) = ^— PR (d z )(e'^J,(z))(0) 

where we extend the function e^" x " on V to a holomorphic function e^ z ^ on the whole 
space Vc- 

The following expansion is a direct consequence of the definition. 
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iilUI|2 



Lemma 8.2. The L-invariant function e 2 ll^ll J\(x) on V has the following expansion in 
terms ofp R (x) 

(8.3) e^^' 2 4(x) = ^j9 11 (x)Cn,,(A), 



where the summation is over all n as determined in (fO) and (fO) 



The next result computes the Fourier transform of the Segal-Bargmann transform Up R ,u 
of p a (A). Denote f(x) 1— > /(A) the Fourier transform on evaluated on a*, 

(8.4) /(A) = I f(x)e l ^ x) dx, AG a*. 

Jv 

Proposition 8.3. The Fourier transform Up R (X) ofUpn is 

(8.5) Upl(X) = (-u)-^(^)ic^(X)e-^ 2 

Proof. The Berezin transform is the convolution operator with Gaussian kernel, whose 
Fourier transform is also a Gaussian. Namely |i?| 2 has e 1 -^ and therefore the Bessel 
function as its eigenf unction. More precisely, we have 

(8.6) \R\ 2 Ux) = ftfe-^' 2 J x (x). 



We may rewrite it as 



27I\ d , _ , , *|U||2, _i||_\||2 



e ^y) e -m\ 2 j x ( x ) d y = (_)l(j A ( a; )ef ll-ll 2 ) e -^IIAH 
v ~ v ~ 

We differentiate both side by the differential operator p R (d x ) and evaluate at x — 0. To do 

this, we observe that 

so that the resulting formula is 

(8-7) (-i/)H [ pn(y)e-^ 2 j,(y)dy = {-^p^A^^ ■ 

Jv v 

The left hand side is actually 

(8-8) (-v) lRl RPn(A) = (-Pp\WJ P 'n(X). 

On the other hand formula the formula Q8.6| ) implies that 

\Wm = ( 2 ^e-^f(X) 



for any / G L 2 (V). Thus 



Substituting this into (8.8) for / = Up R we get 



[-upRpAX) = (-u)^(—)ie^^ 2 U Pn (X). 
~ v ~ 
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The equality ( |8.7| ) becomes now 
namely 

(8-9) C^Pn(A) = (^) l (-^) Hsl |bn||Kn^(A)e-^ IIAl|2 ) 

completing the proof. □ 

The unitarity of U and the Fourier transform implies then the orthogonality relation of 
(n,v Let C\ be the normalization constant so that the following measure is a probability 
measure on a: 



v 

j=l l<i<j<r 



r 

2ia 
'J I 



(8.10) c^VWjJlA.-r 1 II 1^"^ 
for root system of type C, and 

(8.11) C^ie-m? H [Af - AJ 

l<i<j'<r 

for type L\. (Note C\ is independent to the parameter v.) G\ can be evaluated by the 
Selberg-Macdonald formula (proved by Opdam []27|]). 

Corollary 8.4. The W -invariant polynomials ||pn||^Cn,^(A) fo r an orthonormal basis for 
the Hilbert space of W -invariant L 2 -functions with the above probability measure. 



The expansion Q8.3D takes now the following form 
Corollary 8.5. The L-invariant function IWI 2 J\{x) on V has the following expansion in 

Pn(x) 



terms ofpj^x), writing q^ v = nMp the normalized L-invariant polynomial, 



(8.12) e -£llA|| a e SIMI a j (x ) = (^)f V(-l)l^g ni ,(x)^(A). 

n 

9. Segal-Bargmann transform of L-invariant polynomials: bounded 

CASE 

The result is this section is parallel to that of the previous section and is an explicit 
realization of the Corollary We will be brief; see also [ |2~9"| ] and [|2~5|]. 

Recall the Berezin transform B v in (|3.8[). It is proved in [j^] that B v defines a L 7 - 
invariant bounded operator on L 2 (D) with the invariant measure fl3.1|). Let &„ (A) be its 
spectral symbol, namely 

B v (f>\ = h(\)(f)x 

in the sense of spectral decomposition, where 4>\ on D is the spherical function; the function 
6„(A) is explicitly calculated in 
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Define the polynomials £n,i/(A) by the Rodrigues type formula 



£n,i/(A) = I, 1 |i 2 Pu(d x )(h *(x)(f>x(x)) 



x=0 



Thus £n,i/(A) is VT-invariant. 

Theorem 9.1. 77ze L-invariant analytic function h(x, x)~i(f)\(x), when extended to a holo- 
morphic function h(z, z)~%(fi\(z) in a neighborhood of D in V, has the following expansion 
in terms of the L-invariant polynomials Pn(z)> 

(9-1) /1(,Z,Z)-%(Z) = J2tnA±)Pn(z) 

n 



near z = in D. Moreover the spherical transform of the Segal-Bargmann transform ofp n 
is 

(9-2) UpZ(X) = 4^(A)^n,,(A)|bn||^. 



Thus \\p R || u £,n{K)for all n form an orthonormal basis for the space L 2 {a*,C b -^\c{X)\- 2 ) w . 
Conceptually it is better to write ( p.lQ in the form 



(9.3) {R-^iz) = h(z,z)-^x(z) = ^(bn||^n,,(A)) ® 



\Pn\\u 

Note that we have extended the real analytic function h(x,x)~^ to a holomorphic func 

Pn(z) 
llPnlk 



tion, formally written as a, on D. Notice also that both and ||.PnJ|i/£n(A) are 



orthonormal basis in the respective Hilbert spaces. Thus (R~ 1 (p\)(z) is the Schwarz ker- 
nel for the unitary operator from onto L 2 (a*, Co^^-\c(X)\~ 2 ) w obtained by taking the 
composition of the spherical and Segal-Bargmann transforms. 

Remark 9.2. Associated to each root system there are the multi-variable Askey-Wilson 
polynomials determined by the root multiplicities (which are positive real numbers) and 
two more extra parameters []3~3l]. Our polynomial Cn,y(A) depend on extra parameter and 
can be reviewed as some limiting cases of the Askey-Wilson polynomials polynomials. We 
have thus found an alternative simple proof of the their orthogonality relation. Moreover 
the (known) expansion of hr 1 ^. in terms of the Jack symmetric polynomials will also give 
some evaluation formula for the polynomials; we will however not pursue it here. 



Remark 9.3. Consider the spherical function on the Hermitian symmetric space D or on 
its compact dual. There are Jacobi type functions. There have been attempts (see [[[]]) in 
expanding the Jacobi functions in terms of the Jack symmetric polynomials and study the 
combinatorial properties of the coefficients. It turned out (see []37|], p8| ] and [|38|]) however 
that the expansion of the spherical transform multiplied with the reproducing kernel has a 
much better analytical significance. This is also the case here. Further more by considering 
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analytic continuation in the parameter v we may recover the expansion of the spherical 
functions itself. 

Appendix 1 : Evaluation of the constant C 

We consider type D r (r > 3) first, in this case A(x) = 5(x) and the rank r(JD>) is r. 
It is proved in [|T2|], Chapter IV, Exercise C4, that when the //-invariant measure cLl(z) 
on D = H/ K is (regularly) normalized so that 

[ f(z)di(z) = [ /(exp/f) J] (e a ^-e- a ^) ma d n H 

JD Ja+ Q£S+ 

with d n H the regular normalization, then the Plancherel formula reads 



W\ \ D \f{z)\ 2 di{z)= I \f(\)M\)r 2 dn\. 



By regular normalizations d n H on a it is meant that the d n H is the Euclidean measure on 
a induced by the restriction on it of the Killing form on q, multiplied with the factor -j^', 
by the Riesz lemma we get an identification of a* with a and thus similarly get a regular 
measure d n \ on a*. In our case they are 



d n H = , r v 2a(r — 1) dx\ ■ ■ ■ dx r , d n X = , r — = d\\ ■ ■ ■ d\ T 

\/2t7 ~ v 7 ^ \y/2a(r-l) 

if H = + . . . x r £ r G a* and A = Ai/?i H h A r /3 r e a* and the order \ W\ of the Weyl 

group is 2 r ~ 1 r\. There exists now a constant C so that 

I /(^T?7T = C ° / f( z ) d < z ) = Co / /(exptf) J] (e Q W - e - a ^) ma d n H. 
Jd hz{z) J D J a + 

Take /(z) = h(z, z)°~ for sufficiently large a. The left hand side can, by performing Cayley 
transform z i— > ty = 2 = mapping D to its the Siegel domain S (see [|21|], fl39[]), 
be evaluated eventually by the Gindikin Gamma function, while the right hand side is a 
kind of Selberg type integral and is also known. Let us carry out this calculation. Write I\ 
for the left hand side, 



J 1 = / h a -2(z)dm(z) =A r{u -^2 d 



A(w) a 2 c?w 
o'" ~' ' ~ j 5 A(e + M;) 2 ^A( w + e )' 

since the determinant of the differential of the Cayley transform z = 7_i(w) is 

J^iw) =2 d A{e + w)~ 2 *, 

(see [0], Chapter X, Proposition 2.4, for the calculation of the complex Jacobian of the 
Cayley transform), and that 

h(z z) = 4- A{W) 
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This integral over S is evaluated in [03fl, Proposition 4.2, the result is 



h = 4 ^-f) 2 ^4«A-r(,+i)^ J) = J) 

r c (o- + 1) r n (cr + 1) ' 

here, adopting the notation there, r^(x) is the Gindikin Gamma function. The right hand 
side is -j==r \/2a{r — l) r / where 

J = / f[(l-tanh 2 Xj ) a ]J(e x '-^ -e x ^ Xl ) a (e x ^ -e-^-^Ydxi ■ --dx r . 

J xi>x 2 >--->x r >0 - =1 ^ 

Perform change of variables £j = tanhx.,- then dxj = (1 — t|) _1 <itj and 



(9.4) 



(1 -*?)(! 
So that 

/o = ^ r{r ~ l) I n(i - t)y- a ^- 1 n^ 2 - tjydt, 

Jl>t 1 >t 2 >->t r >0 j =1 iK j 

= 4|r(r-D 1 /" JT (1 _ £2 r -a(r-l)-l JT (t 2 _ . . . ^ 

r - •V 1 l r , = 1 i<7 



Changing again variables, letting Sj = t 2 -, the integral becomes 

(9.5) J = 4f^- 1 )2-'-- / JJ(1 - 'i)*-^" 1 II EK* ~ ' ' ' 



its value is 
(9.6) 



2ar(t -i)- r j_ r(or _ a (r _ 1} _ |g _ 1))r( i + a (r _ i) _ |g _ 1)} n ; =i r(f (r - j + 1)) 
r! 11 r^ + i-Kz-i)) r(f)r 



see [|23|], Chapter VI, example 7, pp. 385-386. One of the fraction in the above product is 
the following, and can also be written as 

T(„ - |(r - 1) - |(i - 1)) _ r n ( ff - |(r - 1)) _ T n (o - 



n 



A I> + §-f(i-l)) r n (t7 + |) ro(a + 

this fraction appears also in 1%. Finally the constant Cq is determined by the formula 



1 ~ — f7T~ r 
V Z7T 



and we find that 



_ A d r! r(f) 

^0 — ; r T — ; r V 



/o vW 2-(-D- nj =1 r(f (r - j + l))r(| + f (r - 1) - |(z - 1)) 

The constant for type C r can be evaluated similarly and we leave it to the interested 
reader. 
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Appendix 2: Irreducible real bounded symmetric domains 



We list here the the associated Lie algebras (jj, t) and (h, [) of non-compact irreducible 
bounded symmetric domains D = G/K C Vc and respectively their real bounded symmet- 
ric subdomain D = H/L C V, when V is not an Euclidean Jordan algebra; see [pT|] . They 
were also classified by Olafsson (p5|], 01) through Lie theoretic methods. The restricted 
root systems E of (h, [) are also indicated, the name of the types (Types B, BC, etc) is in 
consistence with Loos j^lf], Proposition 11.18. 

Table 1 . Irreducible real bounded symmetric subdomains 



0,e 




s 


su(r, r), s(u(r) © u(r)) 


so(r, r),so(r) © so(r) 




su(r, r + 6), s(u(r) © u(r + b)) 


so(r, r + 6),so(r) © so(r + 6) 


B r 


su(2r, 2r), s(u(2r) © u(2r)) 


sp(r, r),sp(r) ©sp(r) 


C v 


su(2r, 2r + 26), s(u(2r) © u(2r + 26)) 


sp(r, r + 6),sp(r) ©sp(r + 6) 


BC r 


so*(4r),u(2r) 


so(2r,C),so(2r) 


D 2r 


so*(2r),u(r) 


so(r, C),so(r) 


B T (rodd) 


so(2,p + g),so(2) ©soQo + g)) 


5o(l,p) ©so(l,g),so(p) ffiso(g) 


^ 2 (g^0) 


so(2,p),so(2) ©so(p)) 


so{l,p),so{p) 


Ci 


sp(2r, M),u(2r) 


sp(r,C),sp(r) 


C7 r (6 = 0) 


e 6( _i4), 50(10) +M 


5 p(2,2), 5 p(2)ffisp(2) 




e 6 (_i4), 50(10) +R 


f4(-20),50 (9) 


fid 


e7(-25),e 6 + M 


5U*(8),5 P (4) 


fi 3 
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